-
Notifications
You must be signed in to change notification settings - Fork 87
SDK | Upgrade AWS SDK to v3 - NamespaceS3 #9163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
WalkthroughRemoved two S3 helper exports and migrated S3/STS usage from AWS SDK v2 to v3 across SDK and utility code; NamespaceS3 now requires a top-level bucket argument, replaces .promise() flows with async/await v3 calls, and test coverage for STS v2 was removed while v3 tests remain. (≤50 words) Changes
Sequence Diagram(s)sequenceDiagram
participant App as ObjectSDK
participant NS as NamespaceS3
participant S3 as S3Client (v3)
participant RH as RequestHandler
Note over App: Build v3 s3_params + requestHandler
App->>RH: get_requestHandler_with_suitable_agent(endpoint)
App->>NS: new NamespaceS3({ namespace_resource_id, s3_params, bucket, stats })
NS->>S3: Init S3 client with s3_params + requestHandler
rect rgba(223,242,225,0.6)
Note over App,NS: Read object stream (v3 flow)
App->>NS: read_object_stream(key, range?)
NS->>S3: getObject / headObject (await)
S3-->>NS: Response (metadata + Body stream / error)
NS->>App: Return bridged Readable stream, update stats
end
sequenceDiagram
participant Caller as Cloud Utils Consumer
participant STS as STSClient (v3)
participant CloudUtils as cloud_utils
Caller->>CloudUtils: generate_aws_sdkv3_sts_creds(params)
CloudUtils->>STS: ExecuteRoleWithWebIdentityCommand (await)
STS-->>CloudUtils: { AccessKeyId, SecretAccessKey, SessionToken }
CloudUtils->>Caller: plain credentials object
Caller->>CloudUtils: createSTSS3SDKv3Client(credentials, endpoint)
CloudUtils->>CloudUtils: build S3 client with NodeHttpHandler/requestHandler
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
d72038c
to
9704f13
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 7
🔭 Outside diff range comments (1)
src/sdk/namespace_s3.js (1)
126-132
: Correct pagination field: use NextUploadIdMarker.The response’s next marker is
NextUploadIdMarker
. ReturningUploadIdMarker
will regress pagination.Apply this diff:
return { objects: _.map(res.Uploads, obj => this._get_s3_object_info(obj, params.bucket)), common_prefixes: _.map(res.CommonPrefixes, 'Prefix'), is_truncated: res.IsTruncated, next_marker: res.NextKeyMarker, - next_upload_id_marker: res.UploadIdMarker, + next_upload_id_marker: res.NextUploadIdMarker, };
🧹 Nitpick comments (2)
src/endpoint/s3/s3_utils.js (1)
671-673
: Align error message with v3 shape and consider normalizing the return type.
- The message references "response header" while v3 exposes LastModified as a top-level field. Nit but confusing.
- Optional: consider returning a numeric timestamp for consistency with other paths that use
getTime()
.Apply this diff for the message:
- if (!r.LastModified) throw new Error("date not found in response header"); + if (!r.LastModified) throw new Error("LastModified not found on response");src/sdk/namespace_s3.js (1)
862-896
: Outdated JSDoc types for SSE fields (v2 types).These reference AWS SDK v2 types. Consider updating to v3 inputs for clarity:
- ServerSideEncryption -> import("@aws-sdk/client-s3").ServerSideEncryption
- SSEKMSKeyId -> string, etc.
I can open a quick PR to refresh these annotations if you want.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these settings in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(1 hunks)src/sdk/namespace_s3.js
(32 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (3)
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/sdk/namespace_s3.js (3)
src/sdk/object_sdk.js (3)
noobaa_s3_client
(28-28)require
(27-27)_
(4-4)src/util/cloud_utils.js (5)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)_
(11-11)src/endpoint/s3/s3_utils.js (5)
err
(421-421)err
(445-445)err
(538-538)err
(547-547)_
(4-4)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (1)
src/sdk/object_sdk.js (1)
28-29
: LGTM: importing the v3-aware request handler.This aligns construction with SDK v3 and allows per-endpoint agent selection. Good move.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
src/sdk/object_sdk.js (1)
463-474
: Restore signatureVersion selection to preserve SigV2 endpoints and SDK v2 fallback.Without
signatureVersion
, we effectively force SDKv3/SigV4-only. Deployments/endpoints that still require SigV2 (e.g., resources configured with AWS_V2) will break becausenoobaa_s3_client.should_use_sdk_v2
won’t have a signal to choose SDK v2.Apply this change:
s3_params: { endpoint: r.endpoint, aws_sts_arn: r.aws_sts_arn, + signatureVersion: cloud_utils.get_s3_endpoint_signature_ver(r.endpoint, r.auth_method), credentials: { accessKeyId: r.access_key.unwrap(), secretAccessKey: r.secret_key.unwrap(), }, region: config.DEFAULT_REGION, // SDKv3 needs region forcePathStyle: true, requestHandler: noobaa_s3_client.get_requestHandler_with_suitable_agent(r.endpoint), access_mode: r.access_mode, },
🧹 Nitpick comments (2)
src/sdk/object_sdk.js (2)
28-28
: Nit: Consider lazy-requiring noobaa_s3_client to avoid top-level cost/cycles.object_sdk is a hot path. Requiring the S3 client wrapper at module load time can increase startup cost and risks a circular require with NamespaceS3. Suggest moving this require into the S3 branch of
_setup_single_namespace
(just before creating NamespaceS3) to defer it until needed.
470-473
: Region default and request handler: add safe fallback and ensure handler reuse.
- Region: Some S3-compatible endpoints expect us-east-1 (or ignore region). To reduce signature-mismatch risk when config is unset, prefer a safe fallback.
- Handler: Ensure
get_requestHandler_with_suitable_agent
caches/returns a shared NodeHttpHandler per endpoint (keep-alive, maxSockets) and respects HTTP(S)_PROXY env vars to avoid socket churn and honor proxies.Suggested tweak for region:
- region: config.DEFAULT_REGION, // SDKv3 needs region + region: r.region || config.DEFAULT_REGION || 'us-east-1', // SDKv3 needs region; fall back safely
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
src/sdk/namespace_s3.js
(32 hunks)src/sdk/object_sdk.js
(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- src/sdk/namespace_s3.js
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-jest-unit-tests
- GitHub Check: run-package-lock-validation
🔇 Additional comments (2)
src/sdk/object_sdk.js (2)
466-469
: Credentials nesting matches SDK v3 config expectations.Moving access keys under
credentials
is correct for S3Client v3.
475-475
: Passing bucket as a top-level arg is consistent with the new NamespaceS3 signature.This aligns with the v3 refactor and simplifies
s3_params
.
f5ec83c
to
4b2e592
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (3)
src/endpoint/s3/s3_utils.js (1)
676-678
: Broaden HTTP status acceptance to any 2xx (SDK v3 returns 206 for ranged GET/HEAD).Only allowing 200 will break ranged operations and some HEADs under v3.
Apply this diff:
-function get_http_response_from_resp(res) { - if (res.$metadata.httpStatusCode !== 200) throw new Error("Response return with error"); - return res; -} +function get_http_response_from_resp(res) { + const code = res?.$metadata?.httpStatusCode; + if (!code || Math.floor(code / 100) !== 2) { + throw new Error(`Response returned with error (status ${code})`); + } + return res; +}src/sdk/namespace_s3.js (1)
480-489
: Don’t stringify PartNumberMarker; pass number or undefined.S3 expects a number. Stringifying is unnecessary and can be surprising.
- const req = { + const req = { Bucket: this.bucket, Key: params.key, UploadId: params.obj_id, MaxParts: params.max, - PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker, };src/sdk/object_sdk.js (1)
466-475
: Restore SigV2 support in ObjectSDK S3 client parametersYou’ve removed the
signatureVersion
setting fromsrc/sdk/object_sdk.js
, which means all S3 clients will default to SigV4—even IBM_COS endpoints that require SigV2 will break. Re-add the signature selector based onr.auth_method
:• File:
src/sdk/object_sdk.js
Location: inside thes3_params
object forNamespaceS3
(around line 471)Suggested patch:
forcePathStyle: true, requestHandler: noobaa_s3_client.get_requestHandler_with_suitable_agent(r.endpoint), requestChecksumCalculation: 'WHEN_REQUIRED', + signatureVersion: cloud_utils.get_s3_endpoint_signature_ver(r.endpoint, r.auth_method), access_mode: r.access_mode,
This restores the V2/V4 switch via
cloud_utils.get_s3_endpoint_signature_ver
. If SigV2 is truly deprecated across all pools, remove IBM_COS fromconfig.DEFAULT_S3_AUTH_METHOD
and update docs/tests to reflect a SigV4-only policy.
🧹 Nitpick comments (6)
src/endpoint/s3/s3_utils.js (1)
671-673
: Clarify error and keep semantics consistent (LastModified).The function now relies on LastModified, so the error message should reflect that. Callers currently expect a Date (not a string), which is fine.
- if (!r.LastModified) throw new Error("date not found in response header"); + if (!r.LastModified) throw new Error("LastModified not found in response"); return r.LastModified;src/sdk/namespace_s3.js (5)
32-43
: Use v3 credentials shape when setting access key.With v3 you’re passing credentials under s3_params.credentials, so this.access_key will be undefined and server-side copy checks will always fail.
- this.access_key = s3_params.accessKeyId; + this.access_key = s3_params?.credentials?.accessKeyId || s3_params.accessKeyId;
249-256
: Tighten request typing: only GetObjectRequest is used.JSDoc union includes CopyObjectRequest which isn’t relevant to read path.
- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest | import("@aws-sdk/client-s3").CopyObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest} */ const request = {
265-266
: Superfluous status guard (v3 throws on non-2xx).The extra check is redundant and can hide null-safety issues; removing it simplifies the flow.
- // In v3, non-2xx typically throws; keep this guard harmless. - if (obj_out.$metadata.httpStatusCode >= 300) throw new Error(`S3 getObject failed with status ${obj_out.$metadata.httpStatusCode}`); + // v3 throws on non-2xx; no additional guard needed
361-369
: Head the created version and normalize last_modified_time.Use the returned VersionId (if present) to head the exact created version. Also return epoch millis for consistency with other paths.
- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest} */ - const request = { - Bucket: this.bucket, - Key: params.key, - VersionId: params.version_id, - }; - const res_head = await this.s3.headObject(request); - const last_modified_time = await s3_utils.get_http_response_date(res_head); + /** @type {import("@aws-sdk/client-s3").HeadObjectRequest} */ + const request = { + Bucket: this.bucket, + Key: params.key, + VersionId: res.VersionId || params.version_id, + }; + const res_head = await this.s3.headObject(request); + const last_modified_time = s3_utils.get_http_response_date(res_head)?.getTime?.();
852-866
: Broaden error translation for v3 error shapes (name/$metadata).err.code is often undefined in v3. Use err.name and HTTP status to preserve mappings.
- _translate_error_code(params, err) { - if (err.code === 'NoSuchKey') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err.code === 'NotFound') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err.code === 'InvalidRange') err.rpc_code = 'INVALID_RANGE'; - else if (params.md_conditions) { - const md_conditions = params.md_conditions; - if (err.code === 'PreconditionFailed') { - if (md_conditions.if_match_etag) err.rpc_code = 'IF_MATCH_ETAG'; - else if (md_conditions.if_unmodified_since) err.rpc_code = 'IF_UNMODIFIED_SINCE'; - } else if (err.code === 'NotModified') { - if (md_conditions.if_modified_since) err.rpc_code = 'IF_MODIFIED_SINCE'; - else if (md_conditions.if_none_match_etag) err.rpc_code = 'IF_NONE_MATCH_ETAG'; - } - } - } + _translate_error_code(params, err) { + const code = err?.code || err?.name; + const http = err?.$metadata?.httpStatusCode; + if (code === 'NoSuchKey' || code === 'NotFound' || http === 404) { + err.rpc_code = 'NO_SUCH_OBJECT'; + } else if (code === 'InvalidRange' || http === 416) { + err.rpc_code = 'INVALID_RANGE'; + } else if (params.md_conditions) { + const md_conditions = params.md_conditions; + if (code === 'PreconditionFailed' || http === 412) { + if (md_conditions.if_match_etag) err.rpc_code = 'IF_MATCH_ETAG'; + else if (md_conditions.if_unmodified_since) err.rpc_code = 'IF_UNMODIFIED_SINCE'; + } else if (code === 'NotModified' || http === 304) { + if (md_conditions.if_modified_since) err.rpc_code = 'IF_MODIFIED_SINCE'; + else if (md_conditions.if_none_match_etag) err.rpc_code = 'IF_NONE_MATCH_ETAG'; + } + } + }
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(1 hunks)src/sdk/namespace_s3.js
(32 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (3)
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
- src/test/integration_tests/api/s3/test_s3_ops.js
🧰 Additional context used
🧬 Code Graph Analysis (2)
src/sdk/object_sdk.js (5)
src/sdk/namespace_s3.js (2)
noobaa_s3_client
(14-14)config
(8-8)src/util/cloud_utils.js (1)
noobaa_s3_client
(16-16)src/server/system_services/account_server.js (1)
noobaa_s3_client
(26-26)src/server/bg_services/namespace_monitor.js (2)
noobaa_s3_client
(15-15)config
(13-13)src/sdk/bucketspace_s3.js (1)
noobaa_s3_client
(6-6)
src/sdk/namespace_s3.js (2)
src/sdk/object_sdk.js (3)
noobaa_s3_client
(28-28)require
(27-27)_
(4-4)src/util/cloud_utils.js (5)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)_
(11-11)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-package-lock-validation
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (3)
src/sdk/namespace_s3.js (2)
210-222
: LGTM: v3-compatible InvalidRange handling.Catching by err.name and checking $metadata.httpStatusCode === 416 covers v3 error shapes and keeps the fallback sane.
249-286
: LGTM: streaming fix returns live Node readable and updates stats via tap.This avoids pre-consuming the stream and uses v3’s Body as a Readable directly. Error propagation and rethrow are correct.
src/sdk/object_sdk.js (1)
466-475
: LGTM: v3 client config wiring (credentials, region, forcePathStyle, requestHandler).Solid move to v3 semantics; dynamic agent via requestHandler is appropriate.
Also applies to: 476-478
4b2e592
to
a1dca53
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
♻️ Duplicate comments (2)
src/endpoint/s3/s3_utils.js (1)
670-672
: Accept any 2xx status (v3 returns 206 for ranged responses) and improve error message.Only accepting 200 will incorrectly throw on valid 206 responses (e.g., ranged GET/HEAD). Also tweak the message grammar and add null-safety.
Apply this diff:
-function get_http_response_date(res) { - if (res.$metadata.httpStatusCode !== 200) throw new Error("Response return with error"); - if (!res.LastModified) throw new Error("Date not found in response header"); - return res.LastModified; -} +function get_http_response_date(res) { + const code = res?.$metadata?.httpStatusCode; + if (!code || Math.floor(code / 100) !== 2) { + throw new Error(`Response returned with error (status ${code})`); + } + if (!res?.LastModified) throw new Error("Date not found in response header"); + return res.LastModified; +}src/sdk/namespace_s3.js (1)
480-489
: Don’t stringifyPartNumberMarker
; allow number or undefined. Also note inconsistency with the summary.
- API expects a number;
.toString()
is unnecessary and will throw whennum_marker
is undefined.- The AI summary says this was addressed, but the code still stringifies.
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
🧹 Nitpick comments (3)
src/sdk/namespace_s3.js (3)
24-29
: JSDoc: makebucket
required to match runtime validation.Constructor throws if
bucket
is missing, but the type still marks it optional (bucket?
). Align the doc to avoid confusion.- * bucket?: string, + * bucket: string,
249-257
: Nit: narrow request typing and drop the redundant 2xx guard.
- The request here is only for
getObject
; theCopyObjectRequest
in the union isn’t used.- v3 throws on non-2xx by default—explicit
httpStatusCode >= 300
check is unnecessary.- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest | import("@aws-sdk/client-s3").CopyObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").GetObjectRequest} */ const request = { Bucket: this.bucket, Key: params.key, Range: params.end ? `bytes=${params.start}-${params.end - 1}` : undefined, PartNumber: params.part_number, }; @@ - // In v3, non-2xx typically throws; keep this guard harmless. - if (obj_out.$metadata.httpStatusCode >= 300) throw new Error(`S3 getObject failed with status ${obj_out.$metadata.httpStatusCode}`); + // v3 throws on non-2xx; no explicit httpStatusCode guard needed.Also applies to: 264-266, 279-281
360-369
: Use returned VersionId for HEAD and normalize last_modified_time to a number.
- If versioning is enabled, head the specific version you just wrote (
res.VersionId
) rather thanparams.version_id
.- Normalize
last_modified_time
to a number (ms) for consistency with other code paths that use.getTime()
.- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest} */ - const request = { - Bucket: this.bucket, - Key: params.key, - VersionId: params.version_id, - }; - const res_head = await this.s3.headObject(request); - const last_modified_time = await s3_utils.get_http_response_date(res_head); + /** @type {import("@aws-sdk/client-s3").HeadObjectRequest} */ + const request = { + Bucket: this.bucket, + Key: params.key, + VersionId: res.VersionId, // prefer the just-created version when available + }; + const res_head = await this.s3.headObject(request); + const last_modified_time = s3_utils.get_http_response_date(res_head).getTime();
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(1 hunks)src/sdk/namespace_s3.js
(33 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (3)
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
- src/test/integration_tests/api/s3/test_s3_ops.js
🚧 Files skipped from review as they are similar to previous changes (1)
- src/sdk/object_sdk.js
🧰 Additional context used
🧬 Code Graph Analysis (2)
src/endpoint/s3/s3_utils.js (3)
src/test/integration_tests/api/s3/test_s3_ops.js (9)
res
(109-109)res
(113-113)res
(119-119)res
(174-174)res
(246-246)res
(280-280)res
(309-309)res
(337-337)res
(364-364)src/server/system_services/bucket_server.js (4)
res
(456-457)res
(1307-1307)res
(1437-1439)res
(1907-1910)src/endpoint/s3/s3_bucket_policy_utils.js (3)
res
(132-132)res
(141-141)res
(147-147)
src/sdk/namespace_s3.js (3)
src/sdk/object_sdk.js (4)
noobaa_s3_client
(28-28)require
(27-27)dbg
(9-9)_
(4-4)src/util/cloud_utils.js (6)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)dbg
(3-3)_
(11-11)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-package-lock-validation
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (4)
src/sdk/namespace_s3.js (4)
190-197
: Good v3-compatible InvalidRange handling.Catching by
err.name === 'InvalidRange' || err?.$metadata?.httpStatusCode === 416
and falling back toheadObject
for zero-length objects is correct. Nice.Also applies to: 209-221
434-437
: ETag extraction for multipart copy/upload looks correct.Using
res.CopyPartResult.ETag
for copy andres.ETag
for upload is the right mapping for v3.Also applies to: 471-473
764-775
: Correct API usage for GetObjectAttributes (v3).Typing and request composition look good, and attributes are passed as expected.
817-818
: Nice addition ofObjectSize
as a fallback for size.This aligns with v3’s GetObjectAttributes response shape.
a1dca53
to
c9abf9c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (2)
src/sdk/namespace_s3.js (2)
35-35
: Access key extraction is incorrect for SDK v3 credentialsThe current approach of extracting
accessKeyId
directly froms3_params
will be undefined in SDK v3, as credentials are typically provided unders3_params.credentials
. This could lead to incorrect server-side copy decisions when both namespaces incorrectly compare as undefined.Apply this diff to properly extract the access key:
- this.access_key = s3_params.accessKeyId; + this.access_key = s3_params.credentials?.accessKeyId || s3_params.accessKeyId;Also verify this works with async credential providers if they're used in your setup.
494-494
: Handle undefined num_marker correctlyWhen
params.num_marker
is undefined, calling.toString()
will throw an error. The SDK v3 expects a string forPartNumberMarker
, but it's better to omit the field when undefined.Apply this diff to handle undefined values:
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker ? params.num_marker.toString() : undefined,
🧹 Nitpick comments (1)
src/sdk/namespace_s3.js (1)
365-366
: Fix typo in error messageThere's a typo in the error log message.
- dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and - key : ${params.key}, with erro : `, err); + dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and + key : ${params.key}, with error : `, err);
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(34 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/endpoint/s3/s3_utils.js
- src/util/cloud_utils.js
🚧 Files skipped from review as they are similar to previous changes (1)
- src/sdk/object_sdk.js
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: run-package-lock-validation
- GitHub Check: run-jest-unit-tests
- GitHub Check: Build Noobaa Image
🔇 Additional comments (5)
src/sdk/namespace_s3.js (5)
249-286
: LGTM! Stream handling properly migrated to v3The migration correctly:
- Uses the native Node.js Readable stream from
obj_out.Body
- Returns a live piped stream for the caller to consume
- Properly handles errors and re-throws them
- Updates read statistics via the tap stream
213-219
: LGTM! Proper v3 error handling for InvalidRangeThe error handling correctly checks both the error name and HTTP status code (416) to detect InvalidRange errors in SDK v3, providing proper fallback to
headObject
for zero-length objects.
443-444
: LGTM! Correct ETag extraction for uploadPartCopyThe code correctly extracts the ETag from
res.CopyPartResult.ETag
for copy operations, which is the proper location in SDK v3 responses.
376-376
: LGTM! Proper ETag extraction handling both copy and regular uploadsThe code correctly handles ETag extraction for both regular uploads (
res.ETag
) and copy operations (res.CopyObjectResult?.ETag
).
349-377
: Outdated suggestion – no HeadObject call to avoid
The current implementation doesn’t invokeheadObject
at all. It cleverly captures the response’sDate
header via middleware on the singleputObject
call, so there isn’t a doubled API request. You can safely ignore the recommendation to avoid a non-existentheadObject
call.Likely an incorrect or invalid review comment.
c9abf9c
to
efb51fc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
src/sdk/namespace_s3.js (4)
298-377
: Middleware leak and concurrency hazard when capturing Date on PutObject.
- Adding a named middleware on the shared client and removing it only on success leaks the middleware on failures. Over time you’ll accumulate duplicate middlewares.
- Concurrent uploads share this.s3; using a single static name (“saveDateMiddleware”) risks collisions between requests.
Use a unique name per invocation and remove in finally. Also guard header access and fix a minor typo in the error log. Optional: bridge source_stream errors to the counting stream as well.
- let count = 1; + let count = 1; const count_stream = stream_utils.get_tap_stream(data => { this.stats?.update_namespace_write_stats({ namespace_resource_id: this.namespace_resource_id, bucket_name: params.bucket, size: data.length, count }); // clear count for next updates count = 0; }); - + // propagate upstream errors to the body being sent + params.source_stream.on?.('error', err => count_stream.emit('error', err)); /** @type {import("@aws-sdk/client-s3").PutObjectRequest} */ const request = { Bucket: this.bucket, Key: params.key, Body: params.source_stream.pipe(count_stream), ContentLength: params.size, ContentType: params.content_type, ContentMD5: params.md5_b64, Metadata: params.xattr, Tagging, }; this._assign_encryption_to_request(params, request); - try { - // Add a middleware to log response headers - this.s3.middlewareStack.add( - (next, context) => async args => { - const result = await next(args); - date = new Date(result.response.headers.date); - return result; - }, { - step: "deserialize", // you could also use "deserialize" - name: "saveDateMiddleware", - tags: ["DATE_MIDDLEWARE"], - } - ); - res = await this.s3.putObject(request); - this.s3.middlewareStack.remove("saveDateMiddleware"); - } catch (err) { - dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and - key : ${params.key}, with erro : `, err); + const middlewareName = `saveDateMiddleware:${Date.now()}:${Math.random().toString(36).slice(2)}`; + try { + // Add a per-call middleware to capture the Date response header + this.s3.middlewareStack.add( + (next, _context) => async args => { + const result = await next(args); + const hdrs = result?.response?.headers || {}; + const hdrDate = hdrs.date || hdrs.Date; + if (hdrDate) { + date = new Date(hdrDate); + } + return result; + }, + { step: "deserialize", name: middlewareName, tags: ["DATE_MIDDLEWARE"] } + ); + res = await this.s3.putObject(request); + } catch (err) { + dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and key: ${params.key}, error:`, err); object_sdk.rpc_client.pool.update_issues_report({ namespace_resource_id: this.namespace_resource_id, error_code: String(err.code), time: Date.now(), }); throw err; + } finally { + try { this.s3.middlewareStack.remove(middlewareName); } catch (_) {} }
426-471
: Bridge errors from source_stream in multipart upload body.Same stream error propagation concern applies here. If params.source_stream errors, the Body (count_stream) won’t emit it to callers/S3 client.
let count = 1; const count_stream = stream_utils.get_tap_stream(data => { this.stats?.update_namespace_write_stats({ namespace_resource_id: this.namespace_resource_id, size: data.length, count }); // clear count for next updates count = 0; }); /** @type {import("@aws-sdk/client-s3").UploadPartRequest} */ const request = { Bucket: this.bucket, Key: params.key, UploadId: params.obj_id, PartNumber: params.num, - Body: params.source_stream.pipe(count_stream), + Body: params.source_stream.on?.('error', err => count_stream.emit('error', err)).pipe(count_stream), ContentMD5: params.md5_b64, ContentLength: params.size, };
860-873
: Broaden v3 error translation; err.code is often unset.Many v3 exceptions set name and httpStatusCode instead of code. Current mapping may miss common cases (404/416/412/304), affecting downstream rpc_code and behavior.
Here is a drop-in replacement for _translate_error_code (outside the changed range), preserving existing semantics while adding v3 checks:
_translate_error_code(params, err) { const http = err?.$metadata?.httpStatusCode; const code = err?.code || err?.name; const is404 = code === 'NoSuchKey' || code === 'NotFound' || http === 404; const is416 = code === 'InvalidRange' || http === 416; if (is404) { err.rpc_code = 'NO_SUCH_OBJECT'; return; } if (is416) { err.rpc_code = 'INVALID_RANGE'; return; } if (params.md_conditions) { const md_conditions = params.md_conditions; const is412 = code === 'PreconditionFailed' || http === 412; const is304 = code === 'NotModified' || http === 304; if (is412) { if (md_conditions.if_match_etag) err.rpc_code = 'IF_MATCH_ETAG'; else if (md_conditions.if_unmodified_since) err.rpc_code = 'IF_UNMODIFIED_SINCE'; } else if (is304) { if (md_conditions.if_modified_since) err.rpc_code = 'IF_MODIFIED_SINCE'; else if (md_conditions.if_none_match_etag) err.rpc_code = 'IF_NONE_MATCH_ETAG'; } } }
68-74
: Update the STS S3 client factory call to the correct exportThe
createSTSS3Client
function does not exist incloud_utils.js
; the proper export iscreateSTSS3SDKv3Client
. This mismatch will cause a runtime error.• In
src/sdk/namespace_s3.js
at line 73, change:- this.s3 = await cloud_utils.createSTSS3Client(this.s3_params, additionalParams); + this.s3 = await cloud_utils.createSTSS3SDKv3Client(this.s3_params, additionalParams);
♻️ Duplicate comments (1)
src/sdk/namespace_s3.js (1)
488-497
: Don’t stringify PartNumberMarker; pass number or undefined.The API expects a number. Stringifying is unnecessary and may cause subtle issues. This also reintroduces a previously fixed issue.
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
🧹 Nitpick comments (2)
src/sdk/namespace_s3.js (2)
24-29
: Doc types: make bucket required in the constructor JSDoc.Code enforces a required bucket, but JSDoc marks it optional. Align the type to prevent misuse.
- * s3_params: import("@aws-sdk/client-s3").S3ClientConfig & { + * s3_params: import("@aws-sdk/client-s3").S3ClientConfig & { * access_mode?: string, * aws_sts_arn?: string, * }, - * bucket?: string, + * bucket: string, * stats: import('./endpoint_stats_collector').EndpointStatsCollector, * }} args
32-43
: Access key extraction may be stale with STS/credential providers (affects server-side copy heuristics).You capture access_key only from s3_params.credentials at construction. If STS is used in _prepare_sts_client (or a provider rotates credentials), this.access_key may not reflect the actual key used by this.s3, leading to false negatives/positives in is_server_side_copy.
Consider one of:
- Resolve and cache the current access key from the client provider when needed (e.g., a helper get_current_access_keyId() that awaits this.s3.config.credentials()).
- Or, relax the server-side copy heuristic to attempt CopyObject and fall back on AccessDenied, removing the potentially brittle key comparison.
If you want, I can provide a small helper that memoizes the resolved access key with a short TTL.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(34 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/util/cloud_utils.js
- src/endpoint/s3/s3_utils.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
🧰 Additional context used
🧬 Code Graph Analysis (2)
src/sdk/object_sdk.js (1)
src/sdk/namespace_s3.js (2)
noobaa_s3_client
(14-14)config
(8-8)
src/sdk/namespace_s3.js (3)
src/util/cloud_utils.js (6)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)dbg
(3-3)_
(11-11)src/sdk/object_sdk.js (4)
noobaa_s3_client
(28-28)require
(27-27)dbg
(9-9)_
(4-4)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-package-lock-validation
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (3)
src/sdk/object_sdk.js (1)
28-28
: LGTM: importing noobaa_s3_client for v3 request handlerImport looks correct and aligns with the move to v3 and custom request handlers.
src/sdk/namespace_s3.js (2)
190-221
: LGTM: v3-compatible InvalidRange fallback in read_object_md.Switching to err.name/http 416 detection for InvalidRange is correct for v3 and resolves zero-length object range issues.
808-813
: LGTM: union return typing in _get_s3_object_info is v3-aligned.Accounting for ObjectSize and GetObjectAttributesOutput is good. Size fallback order looks sane.
efb51fc
to
9c1410f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
♻️ Duplicate comments (1)
src/sdk/namespace_s3.js (1)
490-499
: Pass numeric PartNumberMarker; don’t stringify.The API expects a number. Converting to string is unnecessary. Keep it as a number or undefined.
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
🧹 Nitpick comments (6)
src/sdk/namespace_s3.js (5)
24-33
: Makebucket
required in JSDoc to match runtime check.Constructor throws when
bucket
is missing, but the JSDoc marks it optional. Align the types to avoid confusion.- * bucket?: string, + * bucket: string,
249-256
: Narrow the request type for read_object_stream; remove unrelated CopyObject type.This method only performs
getObject
. The union withCopyObjectRequest
(and evenHeadObjectRequest
) is misleading.- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest | import("@aws-sdk/client-s3").CopyObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").GetObjectRequest} */ const request = { Bucket: this.bucket, Key: params.key, Range: params.end ? `bytes=${params.start}-${params.end - 1}` : undefined, PartNumber: params.part_number, };
367-369
: Fix log message typo and improve clarity.Small nit and readability fix.
- dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and - key : ${params.key}, with erro : `, err); + dbg.error( + `upload_object: failed for bucket ${this.bucket}, key ${params.key}, error:`, + err + );
810-816
: Type union references look good for v3 shapes; minor doc nit.Using
import("@aws-sdk/client-s3")._Object
et al. is accurate for v3. Consider avoiding leading-underscore types in JSDoc if you want to keep API surface stable (non-blocking).
888-912
: JSDoc types reference v2 (AWS.S3.*
); update to v3 equivalents.The encryption-related JSDoc still points to v2 types. Replace with v3 S3 shapes (or just document field names) to avoid confusion.
Example:
- * ServerSideEncryption: AWS.S3.ServerSideEncryption, - * SSEKMSKeyId: AWS.S3.SSEKMSKeyId, - * SSECustomerKey: AWS.S3.SSECustomerKey, - * SSECustomerAlgorithm: AWS.S3.SSECustomerAlgorithm, - * CopySourceSSECustomerKey: AWS.S3.CopySourceSSECustomerKey, - * CopySourceSSECustomerAlgorithm: AWS.S3.CopySourceSSECustomerAlgorithm, + * ServerSideEncryption: import("@aws-sdk/client-s3").ServerSideEncryption, + * SSEKMSKeyId: string, + * SSECustomerKey: string, + * SSECustomerAlgorithm: string, + * CopySourceSSECustomerKey: string, + * CopySourceSSECustomerAlgorithm: string,src/sdk/object_sdk.js (1)
472-475
: Optional: clarify interplay with NamespaceS3’s v3-only client creation.NamespaceS3 unconditionally creates a v3 client via
get_s3_client_v3_params
. If any SigV2-only endpoints remain, they won’t be handled by this namespace regardless ofsignatureVersion
. If that’s intentional per PR scope, ignore; otherwise consider routing SigV2 resources to an alternate namespace/client.Would you like me to scan the repo for any remaining AWS_V2/SigV2-only configurations and list buckets/resources that could hit this path?
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(34 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/endpoint/s3/s3_utils.js
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
🧰 Additional context used
🧬 Code Graph Analysis (2)
src/sdk/object_sdk.js (2)
src/sdk/namespace_s3.js (2)
noobaa_s3_client
(14-14)cloud_utils
(10-10)src/util/cloud_utils.js (1)
noobaa_s3_client
(16-16)
src/sdk/namespace_s3.js (2)
src/util/cloud_utils.js (6)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)dbg
(3-3)_
(11-11)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: run-package-lock-validation
- GitHub Check: Build Noobaa Image
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (5)
src/sdk/namespace_s3.js (3)
190-221
: Good v3-compatible InvalidRange fallback and metadata logging.The 416 detection via
err.name
and$metadata.httpStatusCode
is correct for SDK v3, and the inline GET fallback to HEAD for zero-length objects is appropriate.
259-288
: Streaming fix looks good — return a live Readable and propagate errors.Using the Node Readable (
obj_out.Body
), piping through a tap for stats, and rethrowing on errors is the right approach. Error bridging from source to returned stream is also a good safeguard.
377-380
: ETag extraction is correct; date may be undefined for copy path (acceptable).Using
res.ETag || res.CopyObjectResult?.ETag
handles both upload and copy. Returningdate
only for PUT is fine given the constraints.src/sdk/object_sdk.js (2)
28-29
: Importing noobaa_s3_client here is fine; path is consistent with other modules.No issues.
466-476
: s3_params v3 shape looks correct; signatureVersion reintroduced for compatibility.
- credentials nested object, region, forcePathStyle, requestHandler, and requestChecksumCalculation are all appropriate for v3.
- Adding
signatureVersion
viacloud_utils.get_s3_endpoint_signature_ver(...)
restores compatibility with SigV2 endpoints in paths that rely on it.
9c1410f
to
02ff2f8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/sdk/namespace_s3.js (1)
496-497
: Remove unnecessary.toString()
on PartNumberMarker
Theparams.num_marker?.toString()
call is redundant—optional chaining already handlesundefined
, and other numeric params (e.g.MaxParts
) are passed directly. Please update:• File:
src/sdk/namespace_s3.js
, line 496- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(34 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/endpoint/s3/s3_utils.js
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
🧰 Additional context used
🧬 Code Graph Analysis (2)
src/sdk/namespace_s3.js (3)
src/sdk/object_sdk.js (3)
noobaa_s3_client
(28-28)require
(27-27)_
(4-4)src/util/cloud_utils.js (5)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)_
(11-11)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)
src/sdk/object_sdk.js (2)
src/sdk/namespace_s3.js (4)
noobaa_s3_client
(14-14)require
(15-15)config
(8-8)cloud_utils
(10-10)src/util/cloud_utils.js (1)
noobaa_s3_client
(16-16)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-jest-unit-tests
- GitHub Check: run-package-lock-validation
🔇 Additional comments (11)
src/sdk/namespace_s3.js (8)
14-15
: LGTM! SDK v3 migration correctly implemented.The imports are properly structured with the noobaa_s3_client for dynamic request handling and PutObjectCommand for command-scoped middleware usage.
33-45
: LGTM! Constructor properly validates and stores bucket parameter.The bucket validation and explicit bucket storage align well with the v3 migration. The access_key extraction correctly handles the nested credentials structure from v3-style s3_params.
93-93
: Minor: Remove redundant promise() calls in list operations.The list operations have been correctly migrated to v3 style, but ensure all
.promise()
calls have been removed. The current implementation looks correct.
214-222
: LGTM! Improved error handling for InvalidRange with v3 compatibility.The error handling correctly checks both SDK v3 error patterns (
err.name
and HTTP status code) and falls back appropriately for zero-length objects.
250-289
: LGTM! Stream handling properly implemented for v3.The implementation correctly:
- Uses v3
getObject
directly without.promise()
- Returns a live readable stream through the stats tap
- Properly handles error propagation between source and destination streams
- Maintains the streaming nature for efficient memory usage
378-380
: LGTM! Improved ETag handling for both copy and non-copy operations.The ETag extraction correctly handles both
res.ETag
(for PutObject) andres.CopyObjectResult?.ETag
(for CopyObject), and the date capture from middleware is properly returned.
827-827
: LGTM! Object size calculation enhanced for v3.The size calculation correctly includes
res.ObjectSize
in addition to the existing fields, properly handling different response types from v3 APIs.
352-376
: No command-scoped middleware leakageThe
saveDateMiddleware
is added directly to thePutObjectCommand
instance, which is created anew for each request. There’s no modification of the shared S3 client’s middleware stack, so the middleware cannot persist across calls. No further action required.src/sdk/object_sdk.js (3)
28-28
: LGTM! Proper import of noobaa_s3_client for v3 support.The import correctly adds the noobaa_s3_client module needed for dynamic request handler generation in the v3 migration.
461-479
: LGTM! S3 parameters correctly migrated to v3 structure.The s3_params object has been properly updated for SDK v3:
- Credentials are now nested under
credentials
objectregion
is explicitly set with fallback to DEFAULT_REGIONforcePathStyle
replacess3ForcePathStyle
requestHandler
uses the noobaa_s3_client for dynamic agent handlingrequestChecksumCalculation
is properly configuredsignatureVersion
is preserved for compatibility
477-478
: LGTM! Bucket parameter correctly moved to top-level.The bucket is now properly passed as a top-level parameter to the NamespaceS3 constructor, aligning with the v3 migration where bucket is no longer nested in s3_params.
02ff2f8
to
34d1019
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/sdk/namespace_s3.js (2)
821-829
: Potential crash whenres.Metadata
is undefined; avoid mutating source.
_.extend(res.Metadata, ...)
will throw whenMetadata
is undefined (common for list/versions/parts outputs) and mutates the response. Use a fresh object.- const xattr = _.extend(res.Metadata, { - 'noobaa-namespace-s3-bucket': this.bucket, - }); + const xattr = _.assign({}, res.Metadata, { + 'noobaa-namespace-s3-bucket': this.bucket, + });Optionally make
LastModified
parsing defensive:- const last_modified_time = res.LastModified ? res.LastModified.getTime() : Date.now(); + const lm = res.LastModified; + const last_modified_time = lm instanceof Date ? lm.getTime() : (lm ? new Date(lm).getTime() : Date.now());
191-206
: Inline GET path must bufferfirst_range_data
when using v3With v3 the
GetObjectOutput.Body
is a Node.js Readable stream, so settingrequest.Range
alone won’t populatefirst_range_data
. In your current inline‐GET branch you never consume the stream, and later you do:first_range_data: Buffer.isBuffer(res.Body) ? res.Body : undefined,which always yields
undefined
. Callers innamespace_cache.js
and the S3 GET handler rely onfirst_range_data
, so this is now broken.Mandatory fix in
src/sdk/namespace_s3.js
(around lines 191–206): collect up toINLINE_MAX_SIZE
bytes from the stream before returning. For example:if (can_use_get_inline) { request.Range = `bytes=0-${config.INLINE_MAX_SIZE - 1}`; - // fall through without buffering + // v3 returns a Readable; buffer it so first_range_data picks it up + const { Body: stream } = await this.client.send(new GetObjectCommand(request)); + const firstChunk = await stream_utils.collect_stream(stream, config.INLINE_MAX_SIZE); + // override Body so Buffer.isBuffer() sees a Buffer + res.Body = firstChunk; } this._set_md_conditions(params, request);Alternatively, if you only need metadata and no inline bytes, switch to a
HeadObjectCommand
.• Location: src/sdk/namespace_s3.js, inline GET branch (≈191–206)
• Affected behavior:first_range_data
remainsundefined
under v3
• Impacted consumers:
– namespace_cache.js (caching when inline data fits)
– src/endpoint/s3/ops/s3_get_object.js (serving small reads)
♻️ Duplicate comments (4)
src/sdk/namespace_s3.js (4)
214-221
: V3-compatible InvalidRange handling: LGTM.Falling back to
headObject
onerr.name === 'InvalidRange'
orhttpStatusCode === 416
is correct for SDK v3.
270-284
: Error propagation on stream: LGTM.Wiring the source 'error' to the returned stream avoids silent failures. Good fix.
352-366
: Command-scoped middleware for Date capture: LGTM.Moving from client-level to command-level middleware removes race/leak hazards.
491-499
: Do not stringifyPartNumberMarker
; pass number or leave undefined.SDK expects a number. Stringifying is unnecessary and can produce subtle API errors.
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
🧹 Nitpick comments (8)
src/sdk/namespace_s3.js (8)
25-31
: Makebucket
required in JSDoc to match runtime validation.Constructor throws when
bucket
is missing, but JSDoc marks it optional. Align the contract.- * bucket?: string, + * bucket: string,
36-39
: Harden access key extraction for server-side copy checks.In v3, credentials may be provided via a provider function or resolved on the client. Using only
s3_params.credentials?.accessKeyId
can yieldundefined
and cause incorrectis_server_side_copy
decisions.Apply this small reorder to derive from client config as a fallback (and tolerate provider functions):
- this.access_key = s3_params.credentials?.accessKeyId; this.endpoint = s3_params.endpoint; - this.s3 = noobaa_s3_client.get_s3_client_v3_params(s3_params); + this.s3 = noobaa_s3_client.get_s3_client_v3_params(s3_params); + { + const creds = s3_params.credentials ?? this.s3?.config?.credentials; + this.access_key = typeof creds === 'function' ? undefined : creds?.accessKeyId; + }If you intentionally rely only on explicit
s3_params.credentials
, please confirm so we can drop this suggestion.
191-198
: Narrow the request type for clarity.
read_object_md
builds either aHeadObjectRequest
orGetObjectRequest
. Keeping a union in JSDoc is fine, but the current combined type with a single object can confuse tooling.- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest | import("@aws-sdk/client-s3").GetObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & Partial<import("@aws-sdk/client-s3").GetObjectRequest>} */This keeps shared fields while acknowledging optional
Range
.
250-257
: Fix JSDoc type:CopyObjectRequest
is unrelated here.This request is only for
getObject
; includingCopyObjectRequest
is misleading.- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest | import("@aws-sdk/client-s3").CopyObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").GetObjectRequest} */
265-267
: Remove redundant 3xx guard.In v3, non-2xx responses throw. The explicit httpStatusCode >= 300 check won’t be reached and adds noise.
- // In v3, non-2xx typically throws; keep this guard harmless. - if (obj_out.$metadata.httpStatusCode >= 300) throw new Error(`S3 getObject failed with status ${obj_out.$metadata.httpStatusCode}`);
367-369
: Fix typo and improve error log formatting.Minor nit: “erro” → “error”, and avoid multiline template literal with trailing spaces.
- dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and - key : ${params.key}, with erro : `, err); + dbg.error( + `upload_object: Object upload failed for bucket ${this.bucket} and key ${params.key}:`, + err + );
862-876
: Adapt error translation to v3 error shapes.In v3,
err.code
is often unset. Prefererr.name
and$metadata.httpStatusCode
.- _translate_error_code(params, err) { - if (err.code === 'NoSuchKey') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err.code === 'NotFound') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err.code === 'InvalidRange') err.rpc_code = 'INVALID_RANGE'; + _translate_error_code(params, err) { + const name = err?.code || err?.name; + const http = err?.$metadata?.httpStatusCode; + if (name === 'NoSuchKey' || name === 'NotFound' || http === 404) err.rpc_code = 'NO_SUCH_OBJECT'; + else if (name === 'InvalidRange' || http === 416) err.rpc_code = 'INVALID_RANGE'; else if (params.md_conditions) { const md_conditions = params.md_conditions; - if (err.code === 'PreconditionFailed') { + if (name === 'PreconditionFailed' || http === 412) { if (md_conditions.if_match_etag) err.rpc_code = 'IF_MATCH_ETAG'; else if (md_conditions.if_unmodified_since) err.rpc_code = 'IF_UNMODIFIED_SINCE'; - } else if (err.code === 'NotModified') { + } else if (name === 'NotModified' || http === 304) { if (md_conditions.if_modified_since) err.rpc_code = 'IF_MODIFIED_SINCE'; else if (md_conditions.if_none_match_etag) err.rpc_code = 'IF_NONE_MATCH_ETAG'; } } }
879-912
: Update JSDoc encryption types to v3 or simplify to primitives.Current annotations reference v2
AWS.S3.*
types. In v3, these are simple string fields. Simplify to avoid confusion.- /** - * @param {Partial<{ - * ServerSideEncryption: AWS.S3.ServerSideEncryption, - * SSEKMSKeyId: AWS.S3.SSEKMSKeyId, - * SSECustomerKey: AWS.S3.SSECustomerKey, - * SSECustomerAlgorithm: AWS.S3.SSECustomerAlgorithm, - * CopySourceSSECustomerKey: AWS.S3.CopySourceSSECustomerKey, - * CopySourceSSECustomerAlgorithm: AWS.S3.CopySourceSSECustomerAlgorithm, - * }>} request - */ + /** + * @param {Partial<{ + * ServerSideEncryption: import('@aws-sdk/client-s3').ServerSideEncryption | string, + * SSEKMSKeyId: string, + * SSECustomerKey: string, + * SSECustomerAlgorithm: string, + * CopySourceSSECustomerKey: string, + * CopySourceSSECustomerAlgorithm: string, + * }>} request + */
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(34 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/util/cloud_utils.js
- src/endpoint/s3/s3_utils.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
🚧 Files skipped from review as they are similar to previous changes (1)
- src/sdk/object_sdk.js
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/sdk/namespace_s3.js (3)
src/util/cloud_utils.js (6)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)dbg
(3-3)_
(11-11)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)src/sdk/object_sdk.js (4)
noobaa_s3_client
(28-28)require
(27-27)dbg
(9-9)_
(4-4)
}; | ||
this._set_md_conditions(params, request); | ||
this._assign_encryption_to_request(params, request); | ||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@naveenpaul1 did you validate this flow? Does it work as expected and update the counter?
Signed-off-by: Naveen Paul <[email protected]>
34d1019
to
f40f447
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/sdk/namespace_s3.js (2)
56-59
: Make server-side copy decision robust: require real, equal access keys.Avoid permitting server-side copy when both access keys are undefined.
- return other instanceof NamespaceS3 && - this.endpoint === other.endpoint && - this.access_key === other.access_key; + return other instanceof NamespaceS3 && + this.endpoint === other.endpoint && + !!this.access_key && + this.access_key === other.access_key;
862-877
: v3 error mapping misses err.name/http status; improve translate_error_code.In v3, err.code is often unset; names and $metadata.httpStatusCode are reliable. Map 404/416 and precondition errors by name/status.
- const err_code = err.code || err.Code; - if (err_code === 'NoSuchKey') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err_code === 'NotFound') err.rpc_code = 'NO_SUCH_OBJECT'; - else if (err_code === 'InvalidRange') err.rpc_code = 'INVALID_RANGE'; + const http = err?.$metadata?.httpStatusCode; + const err_code = err.code || err.Code || err.name; + if (err_code === 'NoSuchKey' || err_code === 'NotFound' || http === 404) err.rpc_code = 'NO_SUCH_OBJECT'; + else if (err_code === 'InvalidRange' || http === 416) err.rpc_code = 'INVALID_RANGE'; else if (params.md_conditions) { const md_conditions = params.md_conditions; - if (err_code === 'PreconditionFailed') { + if (err_code === 'PreconditionFailed' || http === 412) { if (md_conditions.if_match_etag) err.rpc_code = 'IF_MATCH_ETAG'; else if (md_conditions.if_unmodified_since) err.rpc_code = 'IF_UNMODIFIED_SINCE'; - } else if (err_code === 'NotModified') { + } else if (err_code === 'NotModified' || http === 304) { if (md_conditions.if_modified_since) err.rpc_code = 'IF_MODIFIED_SINCE'; else if (md_conditions.if_none_match_etag) err.rpc_code = 'IF_NONE_MATCH_ETAG'; } }
♻️ Duplicate comments (2)
src/sdk/namespace_s3.js (2)
496-499
: Don't stringify PartNumberMarker; pass number or leave undefined.This was previously flagged; still using toString(). v3 expects a number; optional chaining avoids a throw but the type is still off.
- PartNumberMarker: params.num_marker?.toString(), + PartNumberMarker: params.num_marker,
36-36
: Access key extraction is too narrow; can be undefined in v3 (providers). Gate server-side copy accordingly.In v3, credentials may come from a provider, so s3_params.credentials?.accessKeyId can be undefined. If both sides are undefined, current equality check erroneously allows server-side copy. Store access_key only when known and require truthy keys in the comparison.
- this.access_key = s3_params.credentials?.accessKeyId; + // Only capture a static access key if present; avoid treating "both undefined" as equal later + this.access_key = (s3_params.credentials && 'accessKeyId' in s3_params.credentials) + ? s3_params.credentials.accessKeyId + : undefined;
🧹 Nitpick comments (10)
src/sdk/namespace_s3.js (10)
29-31
: JSDoc mismatch: bucket is enforced by ctor but marked optional.Constructor throws when bucket is missing, so the JSDoc should declare bucket as required for accuracy and tooling.
- * bucket?: string, + * bucket: string,
87-94
: Consider listObjectsV2 to align with AWS recommendations.Not blocking, but v2 API (ListObjectsV2) is the modern, paginated call (ContinuationToken instead of Marker) and generally preferred.
250-257
: Narrow request typing; CopyObjectRequest is unrelated here.This path only calls getObject. Tighten the JSDoc to reduce confusion for maintainers and editors.
- /** @type {import("@aws-sdk/client-s3").HeadObjectRequest & import("@aws-sdk/client-s3").GetObjectRequest | import("@aws-sdk/client-s3").CopyObjectRequest} */ + /** @type {import("@aws-sdk/client-s3").GetObjectRequest} */
367-369
: Typo in error log message.Minor, but noisy in logs.
- dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and - key : ${params.key}, with erro : `, err); + dbg.error(`upload_object: Object upload failed for bucket ${this.bucket} and key: ${params.key}, with error:`, err);
823-826
: Avoid mutating res.Metadata when building xattr.Safer to copy into a new object; prevents accidental mutation and handles undefined metadata.
- const xattr = _.extend(res.Metadata, { - 'noobaa-namespace-s3-bucket': this.bucket, - }); + const xattr = Object.assign( + {}, + res.Metadata || {}, + { 'noobaa-namespace-s3-bucket': this.bucket } + );
234-241
: Include err.name/http status in issues report for v3.err.code may be empty in v3. Include name or http status to make the report useful.
- object_sdk.rpc_client.pool.update_issues_report({ - namespace_resource_id: this.namespace_resource_id, - error_code: String(err.code), - time: Date.now(), - }); + object_sdk.rpc_client.pool.update_issues_report({ + namespace_resource_id: this.namespace_resource_id, + error_code: String(err.code || err.name || err?.$metadata?.httpStatusCode), + time: Date.now(), + });
370-374
: Same as above: enrich error_code for issues report.- object_sdk.rpc_client.pool.update_issues_report({ - namespace_resource_id: this.namespace_resource_id, - error_code: String(err.code), - time: Date.now(), - }); + object_sdk.rpc_client.pool.update_issues_report({ + namespace_resource_id: this.namespace_resource_id, + error_code: String(err.code || err.name || err?.$metadata?.httpStatusCode), + time: Date.now(), + });
474-480
: Same as above: enrich error_code for issues report.- object_sdk.rpc_client.pool.update_issues_report({ - namespace_resource_id: this.namespace_resource_id, - error_code: String(err.code), - time: Date.now(), - }); + object_sdk.rpc_client.pool.update_issues_report({ + namespace_resource_id: this.namespace_resource_id, + error_code: String(err.code || err.name || err?.$metadata?.httpStatusCode), + time: Date.now(), + });
792-799
: Same as above: enrich error_code for issues report.- object_sdk.rpc_client.pool.update_issues_report({ - namespace_resource_id: this.namespace_resource_id, - error_code: String(err.code), - time: Date.now(), - }); + object_sdk.rpc_client.pool.update_issues_report({ + namespace_resource_id: this.namespace_resource_id, + error_code: String(err.code || err.name || err?.$metadata?.httpStatusCode), + time: Date.now(), + });
51-55
: Comment typo.“aboid” -> “avoid”.
- // to aboid ACCESS_DENIED errors. a more complete solution is to always perform the server side copy + // to avoid ACCESS_DENIED errors. a more complete solution is to always perform the server side copy
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
src/endpoint/s3/s3_utils.js
(0 hunks)src/sdk/namespace_s3.js
(35 hunks)src/sdk/object_sdk.js
(2 hunks)src/test/integration_tests/api/s3/test_s3_ops.js
(0 hunks)src/test/unit_tests/util_functions_tests/test_cloud_utils.js
(0 hunks)src/util/cloud_utils.js
(0 hunks)
💤 Files with no reviewable changes (4)
- src/endpoint/s3/s3_utils.js
- src/test/integration_tests/api/s3/test_s3_ops.js
- src/test/unit_tests/util_functions_tests/test_cloud_utils.js
- src/util/cloud_utils.js
🚧 Files skipped from review as they are similar to previous changes (1)
- src/sdk/object_sdk.js
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/sdk/namespace_s3.js (3)
src/util/cloud_utils.js (6)
noobaa_s3_client
(16-16)require
(9-9)require
(13-13)require
(14-14)dbg
(3-3)_
(11-11)src/sdk/object_sdk.js (4)
noobaa_s3_client
(28-28)require
(27-27)dbg
(9-9)_
(4-4)src/server/system_services/bucket_server.js (14)
noobaa_s3_client
(41-41)s3_params
(2048-2058)bucket
(343-343)bucket
(357-357)bucket
(376-376)bucket
(394-394)bucket
(419-419)bucket
(431-431)bucket
(446-446)bucket
(482-482)bucket
(491-491)req
(1116-1116)req
(1149-1149)req
(2013-2013)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build Noobaa Image
- GitHub Check: run-package-lock-validation
- GitHub Check: run-jest-unit-tests
🔇 Additional comments (3)
src/sdk/namespace_s3.js (3)
39-42
: Bucket validation: LGTM.Throwing on missing bucket prevents "undefined" string leakage and downstream flakiness.
191-221
: InvalidRange handling updated for v3: LGTM.Using err.name and 416 http status covers v3 error shapes and correctly falls back to headObject for empty objects.
Also applies to: 222-223
261-289
: Streaming fix and error bridging: LGTM.Using the Node Readable directly, returning a live stream, and propagating source errors addresses the v3 streaming pitfalls.
Describe the Problem
Explain the Changes
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Reference: https://mattsumme.rs/2025/aws-sdk-v3-put-object-stream-regression/
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/migrating/notable-changes/
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-client-s3/Interface/PutObjectOutput/
Summary by CodeRabbit
Refactor
Chores
Tests